Using LSTMs with the subwords dataset¶
In this colab, you'll compare the results of using a model with an Embedding layer and then adding bidirectional LSTM layers.
You'll work with the dataset of subwords for the combined Yelp and Amazon reviews.
You'll use your models to predict the sentiment of new reviews.
import tensorflow as tf
from tensorflow.keras.preprocessing.sequence import pad_sequences
!wget --no-check-certificate \
https://drive.google.com/uc?id=13ySLC_ue6Umt9RJYSeM2t-V0kCv-4C-P -O /tmp/sentiment.csv
import pandas as pd
dataset = pd.read_csv('/tmp/sentiment.csv')
# Extract out sentences and labels
sentences = dataset['text'].tolist()
labels = dataset['sentiment'].tolist()
# Print some example sentences and labels
for x in range(2):
print(sentences[x])
print(labels[x])
print("\n")
Create a subwords dataset¶
We will use the Amazon and Yelp reviews dataset with tensorflow_datasets's SubwordTextEncoder functionality.
SubwordTextEncoder.build_from_corpus() will create a tokenizer for us. You could also use this functionality to get subwords from a much larger corpus of text as well, but we'll just use our existing dataset here.
We'll create a subword vocab_size of only the 1,000 most common subwords, as well as cutting off each subword to be at most 5 characters.
Check out the related documentation for the the subword text encoder here.
import tensorflow_datasets as tfds
vocab_size = 1000
tokenizer = tfds.features.text.SubwordTextEncoder.build_from_corpus(sentences, vocab_size, max_subword_length=5)
# How big is the vocab size?
print("Vocab size is ", tokenizer.vocab_size)
# Check that the tokenizer works appropriately
num = 5
print(sentences[num])
encoded = tokenizer.encode(sentences[num])
print(encoded)
# Separately print out each subword, decoded
for i in encoded:
print(tokenizer.decode([i]))
Replace sentence data with encoded subwords¶
Now, we'll create the sequences to be used for training by actually encoding each of the individual sentences. This is equivalent to text_to_sequences with the Tokenizer we used in earlier exercises.
for i, sentence in enumerate(sentences):
sentences[i] = tokenizer.encode(sentence)
# Check the sentences are appropriately replaced
print(sentences[5])
Final pre-processing¶
Before training, we still need to pad the sequences, as well as split into training and test sets.
import numpy as np
max_length = 50
trunc_type='post'
padding_type='post'
# Pad all sequences
sequences_padded = pad_sequences(sentences, maxlen=max_length,
padding=padding_type, truncating=trunc_type)
# Separate out the sentences and labels into training and test sets
training_size = int(len(sentences) * 0.8)
training_sequences = sequences_padded[0:training_size]
testing_sequences = sequences_padded[training_size:]
training_labels = labels[0:training_size]
testing_labels = labels[training_size:]
# Make labels into numpy arrays for use with the network later
training_labels_final = np.array(training_labels)
testing_labels_final = np.array(testing_labels)
Create the model using an Embedding¶
embedding_dim = 16
model = tf.keras.Sequential([
tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),
tf.keras.layers.GlobalAveragePooling1D(),
tf.keras.layers.Dense(6, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])
model.summary()
Train the model¶
num_epochs = 30
model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
history = model.fit(training_sequences, training_labels_final, epochs=num_epochs, validation_data=(testing_sequences, testing_labels_final))
Plot the accuracy and loss¶
import matplotlib.pyplot as plt
def plot_graphs(history, string):
plt.plot(history.history[string])
plt.plot(history.history['val_'+string])
plt.xlabel("Epochs")
plt.ylabel(string)
plt.legend([string, 'val_'+string])
plt.show()
plot_graphs(history, "accuracy")
plot_graphs(history, "loss")
Define a function to predict the sentiment of reviews¶
We'll be creating models with some differences and will use each model to predict the sentiment of some new reviews.
To save time, create a function that will take in a model and some new reviews, and print out the sentiment of each reviews.
The higher the sentiment value is to 1, the more positive the review is.
# Define a function to take a series of reviews
# and predict whether each one is a positive or negative review
# max_length = 100 # previously defined
def predict_review(model, new_sentences, maxlen=max_length, show_padded_sequence=True ):
# Keep the original sentences so that we can keep using them later
# Create an array to hold the encoded sequences
new_sequences = []
# Convert the new reviews to sequences
for i, frvw in enumerate(new_sentences):
new_sequences.append(tokenizer.encode(frvw))
trunc_type='post'
padding_type='post'
# Pad all sequences for the new reviews
new_reviews_padded = pad_sequences(new_sequences, maxlen=max_length,
padding=padding_type, truncating=trunc_type)
classes = model.predict(new_reviews_padded)
# The closer the class is to 1, the more positive the review is
for x in range(len(new_sentences)):
# We can see the padded sequence if desired
# Print the sequence
if (show_padded_sequence):
print(new_reviews_padded[x])
# Print the review as text
print(new_sentences[x])
# Print its predicted class
print(classes[x])
print("\n")
# Use the model to predict some reviews
fake_reviews = ["I love this phone",
"Everything was cold",
"Everything was hot exactly as I wanted",
"Everything was green",
"the host seated us immediately",
"they gave us free chocolate cake",
"we couldn't hear each other talk because of the shouting in the kitchen"
]
predict_review(model, fake_reviews)
Define a function to train and show the results of models with different layers¶
In the rest of this colab, we will define models, and then see the results.
Define a function that will take the model, compile it, train it, graph the accuracy and loss, and then predict some results.
def fit_model_now (model, sentences) :
model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
model.summary()
history = model.fit(training_sequences, training_labels_final, epochs=num_epochs,
validation_data=(testing_sequences, testing_labels_final))
return history
def plot_results (history):
plot_graphs(history, "accuracy")
plot_graphs(history, "loss")
def fit_model_and_show_results (model, sentences):
history = fit_model_now(model, sentences)
plot_results(history)
predict_review(model, sentences)
Add a bidirectional LSTM¶
Create a new model that uses a bidirectional LSTM.
Then use the function we have already defined to compile the model, train it, graph the accuracy and loss, then predict some results.
# Define the model
model_bidi_lstm = tf.keras.Sequential([
tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(embedding_dim)),
tf.keras.layers.Dense(6, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])
# Compile and train the model and then show the predictions for our extra sentences
fit_model_and_show_results(model_bidi_lstm, fake_reviews)
Use multiple bidirectional layers¶
Now let's see if we get any improvements from adding another Bidirectional LSTM layer to the model.
Notice that the first Bidirectionl LSTM layer returns a sequence.
model_multiple_bidi_lstm = tf.keras.Sequential([
tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(embedding_dim,
return_sequences=True)),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(embedding_dim)),
tf.keras.layers.Dense(6, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])
fit_model_and_show_results(model_multiple_bidi_lstm, fake_reviews)
Compare predictions for all the models¶
It can be hard to see which model gives a better prediction for different reviews when you examine each model separately. So for comparison purposes, here we define some more reviews and print out the predictions that each of the three models gives for each review:
- Embeddings and a Global Average Pooling layer
- Embeddings and a Bidirectional LSTM layer
- Embeddings and two Bidirectional LSTM layers
The results are not always what you might expect. The input dataset is fairly small, it has less than 2000 reviews. Some of the reviews are fairly short, and some of the short ones are fairly repetitive which reduces their impact on improving the model, such as these two reviews:
- Bad Quality.
- Low Quality.
Feel free to add more reviews of your own, or change the reviews. The results will depend on the combination of words in the reviews, and how well they match to reviews in the training set.
How do the different models handle things like "wasn't good" which contains a positive word (good) but is a poor review?
my_reviews =["lovely", "dreadful", "stay away",
"everything was hot exactly as I wanted",
"everything was not exactly as I wanted",
"they gave us free chocolate cake",
"I've never eaten anything so spicy in my life, my throat burned for hours",
"for a phone that is as expensive as this one I expect it to be much easier to use than this thing is",
"we left there very full for a low price so I'd say you just can't go wrong at this place",
"that place does not have quality meals and it isn't a good place to go for dinner",
]
print("===================================\n","Embeddings only:\n", "===================================",)
predict_review(model, my_reviews, show_padded_sequence=False)
print("===================================\n", "With a single bidirectional LSTM:\n", "===================================")
predict_review(model_bidi_lstm, my_reviews, show_padded_sequence=False)
print("===================================\n","With two bidirectional LSTMs:\n", "===================================")
predict_review(model_multiple_bidi_lstm, my_reviews, show_padded_sequence=False)